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making decisions about implementing evidence-based practices 
(EBPs). It walks readers through the process of considering EBPs 
and matching them with the needs of communities, agencies, 
families, and youth. It also presents critical information that will 
help readers understand and use scientific evidence when 
choosing a practice. 
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For additional references on interventions for disruptive behavior 
disorders, see the booklet, Evidence-Based and Promising Practices. 











This KIT is part of a series of Evidence-Based Practices KITs created 
by the Center for Mental Health Services, Substance Abuse and 
Mental Health Services Administration, II.S. Department of Health 
and Human Services. 

This booklet is part of the Interventions for Disruptive Behavior 
Disorders KIT, which includes six booklets: 

How to Use the Evidence-Based Practices KITs 

Characteristics and Needs of Children with Disruptive 
Behavior Disorders and Their Families 

Selecting Evidence-Based Practices for Children with 
Disruptive Behavior Disorders to Address Unmet Needs: 
Factors to Consider in Decisionmaking 

Implementation Considerations 

Evidence-Based and Promising Practices 


Medication Management 


EVIDENCE-BASED 
PRACTICES 

Kfr 

Knowledge Informing Transformation 


Guide to EBPs 


What's in Selecting Evidence- 
Based Practices for Children with 
Disruptive Behavior Disorders 
to Address Unmet Needs: Factors to 
Consider in Decisionmaking 

Introduction to the Six-Step Decisionmaking Process.1 

Case Illustrations.17 

Hints on Understanding Research Study Designs.20 

References.23 


Interventions 
for Disruptive 
Behavior 
Disorders 


















Knowledge Informing Transformation 


Guide to EBPs 


Selecting EBPs 

Introduction to the Six-Step Decisionmaking Process 


This booklet walks readers through the 
process of considering EBPs and matching 
them with the needs of communities, 
agencies, families, and youth. It also 
presents critical information that will help 
readers understand and use scientific 
evidence when choosing a practice. 

The information available for each of the 
18 EBPs defined in Tables 3 and 4 of How 
to Use the Evidence-Based Practices KITs 
can be overwhelming. Even after educating 
yourself about the details of each EBP, 
deciding which EBP to implement in 
an organization and how to implement 
that EBP can be daunting. To help you 
in the selection process, this booklet gives 
you examples of implementation factors 
to be considered at each step in the 
decisionmaking process. Characteristics 
of the EBPs that influence decisionmaking 
are also summarized. 


This booklet can serve as a shortcut 
for indepth browsing through the 18 
EBPs presented in Evidence-Based 
a nd Promising Practices of this KIT 
to narrow the search to match your 
population of interest, your agency, 
and your staff and, most important, to 
satisfy the needs of families and children 
served. Figure 1 presents the six steps in 
selecting a specific EBP or set of EBPs 
to add to a service array: 

■ Determining the evidence for the EBP; 

■ Determining the target population 
of the EBP; 

■ Determining if the outcomes 
of the EBP are meaningful to a 
local population; 
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Figure 1: Decisionmaking in the selection of evidence-based practices 



Adapted from Selecting an evidence-based practice (pp. 1-15), by P. A. Arean and A. Gum, in S. E. Levkoff, H. Chen, J. E. Fisher, & J. S. 
McIntyre (Eds.), Evidence-based behavioral health practices for older adults, 2006, New York, NY: Springer. Copyright 2006 by Springer. 


■ Determining if the practice is consistent with an 
agency’s function; 

■ Determining if implementation is feasible to 
staff; and 

■ Determining if the EBP is acceptable to clients. 

Specific characteristics of the individual EBPs 
are presented in the tables in this booklet. The 
decisionmaking process for selecting an EBP 
entails matching characteristics found in these 
tables with the process illustrated in Figure 1. 

Tables 1A and IB and all of the summary tables 
that follow are organized into the following 
two groups: 

■ EBPs that focus primarily on prevention; and 

■ EBPs that focus on intervention. 


The prevention EBPs are also described in the 
table headings as Prevention/Multilevel because 
many have program components aimed at different 
levels of prevention. For example, some prevention 
programs are considered to be universal because 
they focus on an entire population to prevent 
disruptive behavior disorders. Examples of such 
programs could be outreach or media programs. 

Several other prevention programs are called 
selected because they focus on a specific 
subpopulation to improve behavior problems 
that could turn into disruptive behavior disorders 
among that group of youth. Still others, labeled 
indicated , provide treatment or intervention as 
well as prevention services. 

Intervention usually refers to the treatment of 
a specific disorder, as opposed to the prevention 
of a problem condition. 
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Tables 1A and IB show the level of evidence for 
each EBP and the age, gender, and race/ethnicity 
of the children and adolescents who participated in 
the evaluation of the practice. For example, Table 
1A shows that Project ACHIEVE is a multi-level 
prevention program that has a moderate level of 


evidence to support its effectiveness. Project 
ACHIEVE has been evaluated with boys and girls 
3 to 14 years of age. Approximately half of the 
children who participated in the evaluations were 
White, and half were from diverse populations, 
primarily African American. 


Table 1 A: What is the Level of Evidence for an EBP and is the Population Comparable to Yours? 

— Intervention EBPs 

Triple P-Positive 
Parenting Program 

Level of 
Evidence 

Good support 

Age 

Range 

0-16 

Gender 

Boys | Girls 

B and G 

Race and Ethnicity* 

Groups of children and families in Australia who were primarily White. 

One randomized controlled trial was conducted in China with Chinese children. 

Project ACHIEVE 

Moderate support 

3-14 

B and G 

Evaluation was carried out with groups who were approximately half White 
and half from diverse populations, primarily African American. 

Second Step 

Good support 

4-14 

B and G 

Diverse groups studied: Two studies were conducted primarily with White 
children. In another two studies, the population was primarily African 

American. In one study the proportions of White, African American, and 

Hispanic participants were approximately equal. In another study, the majority 
of participants were African American and secondarily, Hispanic. Another study 
included a small percentage of Asian Americans; and one study was conducted 
in Germany. 

Promoting 

Alternative 

Thinking 

Strategies 

Good support 

5-12 

B and G 

Groups studied were approximately one-half White and one-quarter to one- 
third African American. Asian American, American Indian, and Hispanic 
children combined made up the remainder of the group 

First Steps 
to Success 

Moderate support 

5-6 

B and G 

The children involved in two studies were primarily White. Smaller case studies 
involved primarily African American and some American Indian children with 
minimal participation from Hispanic children. 

Early Risers: 

Skills for Success 

Good support 

6-12 

B and G 

Evaluations included two groups of predominately White children and one 
group of predominately African American children. 

Adolescent 

Transitions 

Program 

Good support 

11-18 

B and G 

Two studies included primarily White children. One study involved primarily 
White and African American children with very small proportions of Hispanics, 
Asian Americans, and American Indians. 


* See Evidence-Based and Promising Practices for more information about the race/ethnicity of the children and adolescents who participated 
in the individual research studies, which established the effectiveness of the EBPs. 
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Table 1B: What is the Level of Evidence and is the Population Comparable to Yours? 
— Intervention EBPs 


Gender 



Level of 
Evidence 

Age 

Range 

Boys | Girls 

Race and Ethnicity* 

Incredible Years 

Good support 

2-12 

B and G 

Four studies have had primarily White participants with no description of other 
ethnic/racial groups. Two studies included African American, Hispanic and other 
multiethnic groups in small proportions. 

Helping the 
Noncompliant 

Child 

Moderate support 

3-8 

B and G 

No specification of ethnicity or race among the studied groups was available. 

Parent-Child 

Interaction 

Therapy 

Good support 

2-7 

B and G 

One study included approximately three-fourths White and one-fourth diverse 
populations (primarily African American). There is support for a culturally 
sensitive adaptation for Puerto Rican and Mexican American families. 

Parent 

Management 

Training-Oregon 

Best support 

4-12 

B and G 

Evaluated primarily on White children and parents. A culturally sensitive 
adaptation of PMTO for Hispanic families has been evaluated as well. 

Brief Strategic 
Family Therapy™ 

Good support 

6-18 

B and G 

Evaluated primarily with Hispanic families. One study's sample was one-fifth 
African American. 

Problem-Solving 
Skills Training 

Good support 

6-14 

B and G 

Studies with groups of approximately three-fourths White and one-fourth 

African American children. 

Coping Power 

Good support 

9-11 

B and G 

Groups studied were approximately half White and half African American 
children. One study was in the Netherlands with Dutch children. 

Mentoring 

Moderate support 

6-18 

B and G 

The major study included a group of approximately three-fourths African 
American children and one fourth Hispanic children. 

Multisystemic 

Therapy 

Best support 

12-18 

B and G 

Most groups that were evaluated were approximately 60% African American 
children and 40% White children, except for two that were approximately 70% 
White and 30% African American. One study included an 84% multiracial 
group of African American and Whites. One study was conducted in Norway 
with Norwegian children. 

Functional Family 
Therapy 

Good Support 

11-18 

B and G 

Groups were predominantly White families. In unpublished studies, diverse 
populations (primarily African American and Hispanic) comprised between 
one fourth and one half of the group. One study was conducted in Sweden. 

Multidimensional 

Treatment 

Foster Care 

Good Support 

3-18 

B and G 

Studies were primarily White children. African American, Hispanic and 

American Indian children were represented in very small proportions. 


* See Evidence-Based and Promising Practices for more information about the race/ethnicity of the children and adolescents who participated 
in the individual research studies, which established the effectiveness of the EBPs. 
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shows that implementing an EBP program within 
a framework of continuous quality improvement 
(CQI) has benefits for all concerned. 


Finally while this KIT focuses on matching EBPs 
with the needs of children, youth, and families and 
the needs of organizations serving them, consider 
the context in which programs are run. Evidence 


EBPs and continuous quality improvement 


Results of state demonstration projects show that implementing EBPs in organizations within a framework 
of continuous quality improvement (CQI) has several benefits: 

■ It builds momentum to get a project off the ground; 

■ It creates the organizational traction needed to achieve broader dissemination of the EBP around 
the state; and 

■ It can provide the justification for sustaining the project. 

Continuous quality improvement principles focus on five areas: 

■ Customer and other stakeholder satisfaction with the quality and outcomes of services; 

■ Employee and customer empowerment to identify problems, identify opportunities for improved 
care, and take necessary action; 

■ The identification of organizational processes and systems, not individuals, as the source of problems; 

■ The use of structured problem-solving approaches based on data analysis; and 

■ The use of inclusive cross-functional teams (Shortell et al., 1995). 

In a CQI framework, the needs of the child and family are comprehensively assessed and carefully 
matched with services and treatments. Outcomes are routinely monitored to ensure that the services 
and treatments are producing the desired results. Efforts to improve and enhance services and 
treatments are implemented as needed to improve outcomes for children and families. 
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Step 1 


What is the evidence 
for a practice? 

The first step shown in Figure 1 is to determine 
how much evidence supports an intervention. 

As examples, two categories are in the left boxes: 
Best Support and Good Support. Other levels of 
support are also possible. Generally, a higher level 
of evidence is desirable. 

This booklet uses Hawaii’s system of rating EBPs, 
because it is based on the criteria used by the 
American Psychological Association but integrates 
a broader range of evidence, with five categories: 

■ Best support; 

■ Good support; 

■ Moderate support; 

■ Minimal support; and 

■ Known risks. 


Categories are based on the type and amount 
of rigorous scientific study that a practice has 
undergone. The following outline presents the 
criteria of the system’s ratings. Find the ratings 
with explanatory material at http://hawaii.gov/health/ 
mental-health/camhd/library/pdf/ebs/ebsOI 1 .pdf. 

The EBPs covered in the KIT have levels of 
evidence of at least moderate support, while most 
have good support. Using Hawaii’s system of rating, 
see that most interventions in Tables 1A and IB 
are at the second level because they have undergone 
rigorous testing but only by one group of researchers. 
In other words, separate, independent researchers 
have not replicated the findings, either because no 
independent research studies have been completed 
or because independent studies did not confirm 
the results of earlier studies. 


Criteria for level of evidence (Hawaii EBP Services Committee, 2004) 


Level 1: Best Support 

I. At least two good between-group design experiments demonstrating efficacy in one 
or more of the following ways: 

a. Superior to pill placebo, psychological placebo, or another treatment 

b. Eguivalent to an already established treatment in experiments with adeguate statistical 
power (about 30 per group; cf. Kazdin & Bass, 1989) 

OR 

II. A large series of single case design experiments (n > 9) demonstrating efficacy. 

These experiments must have: 

a. Used good experimental designs 

b. Compared the intervention to another treatment as in l.a. 

AND 

Further criteria for both I and II: 

II. Experiments must be conducted with treatment manuals. 

III. Characteristics of the client samples must be clearly specified. 

IV. Effects must have been demonstrated by at least two different investigators or teams 
of investigators. 
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Criteria for level of evidence (Hawaii EBP Services Committee, 2004) 


Level 2: Good Support 

I. Two experiments showing the treatment is superior (statistically significantly) to a waiting- 
list control group. Manuals, specification of sample, and independent investigators are 
not required. 

OR 

II. One between-group design experiment with clear specification of group, use of manuals, 
and demonstrating efficacy by either: 

a. Superior to pill placebo, psychological placebo, or another treatment 

b. Equivalent to an already established treatment in experiments with adequate statistical 
power (about 30 per group; cf. Kazdin & Bass, 1989) 

OR 

III. A small series of single case design experiments (n > 3) with clear specification of group, 
use of manuals, good experimental designs, and compared the intervention to pill or 
psychological placebo or to another treatment. 

Level 3: Moderate Support 

I. One between-group design experiment with clear specification of group and treatment 
approach and demonstrating efficacy by either: 

a. Superior to pill placebo, psychological placebo, or another treatment 

b. Equivalent to an already established treatment in experiments with adequate statistical 
power (about 30 per group; cf. Kazdin & Bass, 1989) 

OR 

II. A small series of single case design experiments (n > 3) with clear specification of group 
and treatment approach, good experimental designs, at least two different investigators 
or teams, and comparison of the intervention to pill, psychological placebo, or 
another treatment. 

Level 4: Minimal Support: Treatment does not meet criteria for Levels 1, 2, 3, or 5. 

Level 5: Known Risks 
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Step 2 


Is the study population 
comparable to yours? 

Step 2 is relatively straightforward. Stakeholders 
must decide if an intervention is appropriate for 
the population they serve. 


Even if an intervention has not been studied for 
use with a particular population, it doesn’t 
necessarily follow that the EBP will be ineffective 
with that population. The only conclusion that can 
be drawn is that no current evidence shows that it 
is effective with that population. See the following 
note on race and ethnicity. 


Race and Ethnicity 


It is important to mention here that the issue of race and ethnicity is quite controversial because children 
and families of diverse races and ethnicities frequently are represented in very small proportions, if at all, 
in studies evaluating the effectiveness of interventions. This issue is particularly relevant for Hispanic, 
American Indian, and Asian groups. 

Isaacs, Huang, Hernandez, and Echo-Hawk (2005) suggest two approaches for ensuring culturally 
competent EBPs for children and families of color: 

■ Cultural adaptations of existing evidence-based practices; and 

■ Use of culturally specific interventions. 

It is beyond the scope of this KIT to provide information about practice-based evidence models that may 
be used across many different cultures for working with children and youth with Disruptive Behavior 
Disorders. However, see Isaacs et al. (2005) for a comprehensive discussion of the issues. 

Also, obtain information at the Portland Research and Training Center about a current project to develop 
and test practice-based evidence approaches to establish the effectiveness of programs and services, 
including culturally specific practices. 

See http://www.rtc.pdx.edu/ 
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Step 3 


Are outcomes meaningful 
to a local population? 


Step 4 


How does a practice fit 
with an agency? 


The next step in the process is to answer 
the question, Are the outcomes of the EBP 
meaningful to my agency and the children and 
families we serve? Some of the more commonly 
desirable outcomes for children and adolescents 
with Disruptive Behavior Disorders include 
the following: 

■ Reduction in: 

□ Aggressive behavior; 

□ F amily conflict; 

□ School absences and failure; and 

□ Legal system involvement. 

■ Increase in: 

□ School achievement; 

□ Positive peer relationships; 

□ Parenting skills; and 

□ Ability to access other services. 

Tables 2A and 2B provide a quick reference 
to the outcomes that have been seen in the 
evaluation studies conducted on the EBPs 
included in this KIT. 


Adding EBPs to existing service arrays often 
requires, at a minimum, carefully examining staffing 
patterns, staff training and supervision, procedures 
for measuring and monitoring treatment fidelity 
and outcomes, and financing methods. 

Unless agencies are already thoroughly engaged 
in valuing and using data for continuous quality 
improvement, most agencies will have to commit 
to change. This w ill entail building an infrastructure 
to accommodate and support evidence-based 
decisionmaking and EBPs. 

Therefore, the closer the fit between the 
characteristics of an EBP with an agency’s mission 
and functions, the easier the accommodation may 
be for the agency. 

Evidence-Based and Promising Practices in this 
KIT contains extensive descriptions of all 18 of the 
EBPs with detailed information about characteristics 
of the EBPs, training requirements, and specifics 
about how these EBPs have been financed. Tables 
3A, 3B,4A,4B, 5A, and 5B in this booklet provide 
an overview of characteristics of EBPs that help 
determine if they are good fits with an agency. 
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Table 2A: Are the Prevention/Multilevel Outcomes Meaningful to a Local Population 


Intervention 

Outcomes 

Triple P-Positive Parenting Program 

■ Increase in parental confidence 

■ Reduction in child behavior problems 

■ Improvements in dysfunctional parenting styles 

Project ACHIEVE 

■ Decrease in discipline problems 

■ Decrease in special education referrals and placements 

■ Increase in positive school climate 

■ Improvements in academic achievement 

Second Step 

■ Increase in prosocial behavior and social reasoning 

■ Improvement in self-regulation of emotions 

■ Decreased verbal and physical aggression 

■ Decrease in problem behaviors 

Promoting Alternative Thinking Strategies 

■ Increase in ability to label feelings 

■ Reductions in classroom aggression 

■ Increase in self-control; 

■ Decrease in teacher-reported internalizing and externalizing behaviors 

First Steps to Success 

■ Decrease in aggression 

■ Increase in time spent on academics 

■ More positive behavior demonstrated 


Table 2B: Are the Intervention Outcomes Meaningful to a Local Population? 

Intervention 

Outcomes 

Incredible Years 

■ 

■ 

■ 

Increases in parents' use of effective limit setting, nurturing, and supportive parenting 
Improvement in teachers' use of praise 

Reductions in conduct problems at home and school 

Helping the Noncompliant Child 

■ 

■ 

Improvement in parenting skills 

Improvement in child's behavior and compliance 

Parent-Child Interaction Therapy 

■ 

■ 

Improvement in parent-child interaction style 

Improvement in child behavior problems 

Parent Management Training-Oregon 

■ 

■ 

■ 

Significant reductions in child's behavioral problems 

Reductions in coercive parenting 

Increases in effective parenting 

Brief Strategic Family Therapy™ 

■ 

■ 

■ 

■ 

■ 

Decrease in substance abuse 

Improved engagement in therapy 

Decrease in problematic behavior 

Increased family functioning 

Decrease in socialized aggression and conduct disorder 

Problem-Solving Skills Training 

■ 

■ 

Improvement in behavior as rated by teachers and parents 

Family life-functioning improvement 

Coping Power 

■ 

■ 

■ 

Decrease in substance abuse 

Improvement in social skills 

Less aggressive belief system 

Mentoring 

■ 

■ 

■ 

Increased confidence in school performance 

Improved family relationships 

Increased prosocial behaviors 

Multisystemic Therapy 

■ 

■ 

■ 

■ 

■ 

Decreased arrests and re-arrests 

Increased school attendance 

Decreased behavior problems 

Decreased substance use 

Improved family relations 

Functional Family Therapy 

■ 

■ 

Reduction in recidivism and out-of-home placements 

Improvements in family communication style, family concept, and family interaction 

Multidimensional Treatment Foster Care 

■ 

■ 

■ 

■ 

Fewer runaways 

Less chance of arrest or decrease in arrest rates 

Decrease in violent activity involvement of incarceration after completing program 

Fewer permanent placement failures. 
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Tables 3A and 3B summarize key features of the 
various EBPs. They include the following: 

■ The setting for prevention/multilevel or 
intervention programs; 

■ The length of prevention/multilevel or 
intervention programs; 


■ Whether a family component exists; 

■ Who delivers the programs; and 

■ The format of the EBP (individual or 
group sessions). 


Table 3A: Fit with Agency: 

Prevention/Multilevel EBPs 




Setting 

Clinic Home 
or School 

What is its length? 

Family component 

Who delivers? 

Format 

Individual 
or Group? 

Triple P-Positive 
Parenting Program 

C, H, S 

Varies due to level 
implemented (from 1-2 
sessions to 8-10 sessions) 

Parent training, home 
visits, partner support 
skills, mood management 
workbook material 

Trained mental health 
professionals, health care 
professionals and school staff 
(counselors, parent liaisons) 

1, G 

Project ACHIEVE 

S 

3 years 

Parent training 

School administrators, 
teachers and 
chosen facilitators 

G 

Second Step 

S 

School year 

Family guide that includes a 
video-based parent training 
program to help parents 
reinforce skills at home 

Classroom-based intervention 
implemented by teachers 
and counselors 

G 

Promoting 

Alternative 

Thinking 

Strategies 

S 

5 years, 3 times a week for 
20-30 minutes 

None 

Teachers and counselors. It 

is recommended to hire a 

PATHS coordinator. 

G 

First Steps 
to Success 

H, S 

3-4 months 

Parent training delivered in 
the home 

Coaches with MA degree 
plus clinical experience 
work alongside teachers and 
parents/guardians 


Early Risers: 

Skills for Success 

H, S 

3-6 months for recruitment/ 
screening 2-3 years for 
the intervention 

Parent education workshops, 
individualized family support 

Specially trained 
family advocate 

I 

Adolescent 

Transitions 

Program 

S 

Varies by level: 

Level 1: 6 weeks 

Level 2: 3 sessions 

Level 3: 12 sessions 

with 3-month followup 

Family management groups, 
individual family therapy 

Master's level counselors 

1, G 
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Table 3B: Fit with Agency: 

Intervention EBPs 





Setting: 

Clinic Home 
or School 

What is its length? 

Family component 

Who delivers? 

Format: 

Individual 
or Group? 

Incredible Years 

H, S 

Less than 22 weeks 

Parent training 

Parents, teachers, counselors, 
social workers or master's 
level therapists 

G 

Helping the 
Noncompliant 

Child 

C, H 

8-10 sessions 

Parent training 

Master's level therapist 

1 

Parent-Child 

Interaction 

Therapy 

C 

10-16 sessions 

Parent training, coaching 

Master's or doctoral level 
therapist 

1 

Parent 

Management 

Training-Oregon 

C, H 

20 sessions over 13 months 

Parent training 

Trained master's level 
therapist 

1 

Brief Strategic 
Family Therapy™ 

C, H 

12-16 sessions over 3 months 

Family therapy 

Master's or doctoral level 
therapist 

1 

Problem-Solving 
Skills Training 

C, H 

20 sessions 

Parent training 

Master's level therapist 

1 

Coping Power 

S 

15-18 months 

Parent training 

Program specialist/master's or 
doctoral level therapist and 
school guidance counselor 

G 

Mentoring 

H, S 

1 year or longer 

None 

Trained adults 

1 

Multisystemic 

Therapy 

H, S 

3-5 months 

Family therapy, parent 
training 

Master's or doctoral level 
therapist 

1 

Functional Family 
Therapy 

C, H 

8-12 sessions 

Family therapy 

Paraprofessionals and 
master's level therapists 

1 

Multidimensional 

Treatment 

Foster Care 

C, H, S 

6-9 months 

Training, weekly meetings 

Trained treatment families 

1 
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Tables 4A and 4B address training the workforce 
in the skills and competencies required for the 
various EBPs. They cover the following: 

■ The developer’s involvement in training; 

■ Location of training; 


■ Length of training; 

■ Cost of training; and 

■ Availability of followup coaching. 


Table 4A: Fit with Agency: Training and Coaching/Consultation — Prevention/Multilevel EBPs 

Triple P- Positive 
Parenting Program 

Training by 
developer? 

Yes 

Where? 

Onsite and Regional 

Length of training? 

2 sets of 2-3 days with repeat 
in 8-10 weeks 

Cost? 

$21,000 per 20 trainees 

Is followup 

coaching 

available? 

Yes 

Project ACHIEVE 

Yes 

Onsite 

YR 1:5-8 days 

YR2:4-8 days 

YR3:4-6 days 

Average of $25,000 per year to 
$75,000 for 3 years 

Yes 

Second Step 

Yes 

Onsite or offsite 
options 

1'k days or 1-day option 

Options: 

■ $399-$499 per person off 
site 

■ Onsite for $6,475 + travel 
for up to 40 people 

■ $1,600 1-day version onsite 

Yes 

Promoting 

Alternative 

Thinking 

Strategies 

Indirectly 

Onsite 

2-3 days 

Options: 

■ 1 trainer, 2 days, and 30 
participants and ongoing 
technical assistance (TA) for 
$4,000-$5,000 plus travel 

■ 1 trainer, 2 days, and 30 
participants but no ongoing 

TA for $3,000 + travel 

Yes 

First Steps to 
Success 

Indirectly 

Onsite 

2 days for consultants/ 
caseworkers, 1 day for teachers 

$1,000-1,500 per day plus travel 
expenses for up to 30 coaches 
and 50 teachers 

Yes 

Early Risers: Skills 
for Success 

Yes 

Onsite 

4 days 

$5,000-8,000 

Yes 

Adolescent 

Transitions 

Program 

Yes 

Onsite 

Stage 1: 4-5 days 

Stages: 2 and 3 varies 

Varies by stage and size of group 
from $500-1,850 + materials 

Yes 
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Table 4B: Fit with Agency: Training and Coaching/Consultation—Intervention EBPs 


Incredible Years 

Training by 
developer? 

Yes 

Where? 

Onsite and offsite 

Length of training? 

2-3 days per curriculum 
(3 possible curricula in total) 

Cost? 

$300-400 per person offsite 

$1,500 per day + travel 
expenses onsite 

Is followup 

coaching 

available? 

Yes 

Helping the 
Noncompliant 

Child 

Yes 

Onsite 

2 days minimum 

$1,500 per day + expenses 

Yes 

Parent-Child 

Interaction 

Therapy 

Yes 

Offsite 

5 days 

$3,000 

No 

Parent 

Management 

Training-Oregon 

Yes 

Onsite 

18 workshop days spread over 

1 year 

$25,000 per trainee 

Yes 

Brief Strategic 
Family Therapy™ 

Yes 

Onsite 

4 (3-day) workshops 

$60,000 (includes coaching) 

Yes 

Problem-Solving 
Skills Training* 

Yes 

Onsite 

6-12 months 

Graduate school tuition 

No 

Coping Power 

Yes 

Onsite 

3 days 

$5,000 + travel expenses 
and materials 

Yes 

Mentoring 

No 

Regional 

Varies by model 

Varies by model; some free 

Yes 

Multisystemic 

Therapy 

Yes 

Regional 

5 days for staff; 2 days for 
supervisors 

$26,000 for a team of 

4-6 staff members 

Yes 

Functional Family 
Therapy 

Yes 

Onsite and offsite 

2 days onsite plus 

2 days offsite. 

Followup training of 3 onsite 
visits per year, 2 days each 

For 3-8 therapists, about 
$35,000 Year 1; 

$18,000 Year 2 

Yes 

Multidimensional 

Treatment 

Foster Care 

Yes 

Onsite and offsite 

4-5 days for staff; 2 days for 
treatment parents 

$40,000-$50,000 per site 

Yes 


* Graduate students have been trained as therapists as part of research studies. An infrastructure for training other clinicians is in the planning stages. 
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The final tables that refer to Fit with Agency are 
Tables 5A and 5B on Monitoring and Financing 
Options. An integral part of using EBPs within a 
CQI framework is measuring and assessing the 
fidelity of the interventions (that is, the extent to 
which the treatment is delivered as intended) and 
the client outcomes that result from treatment. 
The availability of measurement instruments 
facilitates these processes. 


Tables 5A and 5B provide brief information about 
whether such instruments are available from the 
developer or purveyor of the various EBPs, and the 
developers’ expectations about their ongoing 
measurement. Tables 5A and 5B also provide 
information obtained from the developers of the 
EBPs related to how the EBPs have been financed. 


Table 5A: Fit with Agency: Monitoring and Financing Options—Prevention/Multilevel EBPs 

Triple P-Positive 
Parenting Program 

Is there a fidelity/ 
adherence measure? 

Yes 

If Yes, What Is the 
expectation of use? 

Not required 

Is an outcome 
measure specified? 

Yes 

Financing options 

Grants, State Funds 

Project ACHIEVE 

Yes 

Required 

Yes 

Special Education Funds, School 
Improvement Funds, Safe School 
Grants, Foundation, Partial Medicaid 

Second Step 

Yes 

Not required 

Yes 

Safe and Drug Free Schools 

Promoting Alternative 
Thinking Strategies 

Yes 

Not required 

Yes 

Safe and Drug Free Schools, 

School Board Funds, Grants 

First Steps to Success 

Yes 

Required 

Yes 

School Districts, Grants 

Early Risers: 

Skills for Success 

Yes 

Not required 

Yes 

Local Grants, County Funds 

Adolescent Transitions 
Program 

Yes 

Required 

Yes 

Federal Grants 


Table 5B: Fit with Agency: Monitoring and Financing Options—Intervention EBPs 


Is there a fidelity/ 
adherence measure? 

If Yes, What Is the 
expectation of use? 

Is an outcome 
measure specified? 

Financing options 

Incredible Years 

Yes 

Not required 

Yes 

Grants, State Funds 

Helping the 
Noncompliant Child 

Yes 

Not required 

Yes 

Grants, State, Private Insurance, 
Medicaid 

Parent-Child 

Interaction Therapy 

Yes 

Not required 

Yes 

Grants, State, Private Insurance, 
Medicaid 

Parent Management 
Training-Oregon 

Yes 

Required 

Yes 

Grants, State, Private Insurance, 
Medicaid 

Brief Strategic 

Family Therapy™ 

Yes 

Required 

Yes 

Grants, State, Private Insurance, 
Medicaid 

Problem-Solving 

Skills Training 

Yes 

Not required 

No 

Grants, State, Private Insurance, 
Medicaid 

Coping Power 

Yes 

Not required 

No 

Safe and Drug Free Schools, Local 
Grant Funding 

Mentoring 

Yes 

Not required 

Yes 

Grants, Medicaid 

Multisystemic Therapy 

Yes 

Required 

Yes 

Grant, State, Medicaid 

Functional Family 
Therapy 

Yes 

Required 

Yes 

Grant, State, Medicaid 

Multidimensional 
Treatment Foster Care 

Yes 

Required 

Yes 

Grant, State, Medicaid 
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How does a practice 
fit with staff? 

Another important piece to consider when 
selecting an EBP is its fit with an organizations 
clinicians and other staff members. Gregory 
Aarons’ (2004, 2005) study on provider attitudes 
about implementing EBPs has shown that the 
following factors effect an EBPs fit: 

■ The appeal of the specific EBP itself; 

■ The requirement to use an EBP; 

■ The openness of the provider to new practices; 
and 

■ The perceived difference between usual 
practices and an EBP (Aarons, 2005). 

How does a practice 
fit with youth and family? 

Families and youth are driving changes to systems. 
These mental health services should: 

■ Be culturally sensitive; 

■ Allow for shared decisionmaking; 

■ Incorporate strength-based principles; and 

■ Respect each individual family member’s voice. 


Additionally, their selection of EBPs may 
be dependent on several factors, such as 

■ Presenting problems or diagnosis; 

■ Access to care; 

■ Availability of care; 

■ Personal choice; and 

■ Cost. 

Difficulty accessing services and the limited 
availability of services affect the experience 
of families and youth in mental health systems. 
Insurance coverage and transportation availability 
can present more barriers. 

Michigan’s Association for Children’s 
Mental Health guide for families Evidence- 
Based Practice — Beliefs, Definitions, 

Suggestions for Families (2004) is a helpful 
resource for families and youth preparing 
for meetings with care providers. (See 
http://www.acmh-mi.org/41447_ACMH_Booklet.pdf.) 

The family worksheets accompanying this KIT 
contain sample questions that families may ask 
to assess the fit of EBPs with their own needs 
and circumstances. 
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Case Illustrations 


The following two case illustrations provide examples of the process of selecting EBPs. 
One focuses on prevention, the other on intervention. 


Case Illustration 1: Prevention/Multilevel 

The Case of Oyster Elementary Public School 

As principal of Oyster Elementary Public School, I have 
noticed a growing need for mental health services for 
children in our school. In the last few years, the number 
of children who demonstrate verbally and physically 
aggressive behaviors, limited social reasoning, and 
an inability to manage emotions has increased. 

At the last school staff meeting, teachers and school 
counselors, growing increasingly frustrated with negative 
classroom behaviors, asked for leadership on how to 
respond in an effective, unified way to meet the needs 
of the children. 

An increasing number of parents have also approached 
me with concerns about their child being bullied at 
school or on the bus. 

Consultation 

After speaking with my school superintendent, I 
consulted with our state project director of mental 
health services, Dr. Jones. I asked her for information 
on possible programs to implement in our school to 
prevent the negative child behaviors. Dr. Jones directed 
me to the resource, A Guide for Selecting and Adopting 
Evidence-Based Practices for Children and Adolescents with 
Disruptive Behavior Disorder. 

She informed me that the guide included 18 practices, 
7 of which fit in the category of Prevention/Multilevel 
Practices that were primarily implemented in schools. 
Dr. Jones also suggested that I bring together an 
advisory panel for the process of selecting and 
implementing a program. 


The advisory panel 

Following her advice, I brought together an advisory 
panel that consisted of the following people: 

■ The state children's mental health director; 

■ The director of a community mental health center; 

■ The superintendent of the school district; 

■ The school district psychologist; 

■ The director of special education at the school; 

■ The director of guidance counseling from the school; 

■ Two classroom teachers; 

■ A group of parents; and 

■ Myself. 

The panel agreed to commit themselves to a series of 
meetings for the selection and implementation process. 

The following process took place over a number 
of months. 

Considering needs of audience of interest 

The first thing that we explored were the demographics 
of the children at our school: boys and girls at the K-5 
grade level, from White, African American, and Hispanic 
backgrounds. The KIT (see Table 1A) provided guick 
access to prevention programs that, at first glance, were 
a match with our school community: Project ACHIEVE, 
Second Step, and Promoting Alternative Thinking 
Strategies (PATHS). 
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Considering desired outcomes 

Then, we explored how meaningful each programs' 
expected outcomes and levels of evidence were to 
Oyster Elementary's students. We decided that we 
would form focus groups to gather formal information 
from the school community about desired outcomes 
from an intervention. We conducted the focus groups 
and then matched the information gathered with 
the program outcomes identified within the KIT (see 
Table 1 A). By doing this, we realized that the choice of 
prevention programs that now seemed most appropriate 
to consider narrowed to two: Second Step and PATHS. 

Considering fit with the school 

Next, we explored together how well each program 
would fit with our school. Through our discussion, 
it became clear that Oyster is a school where teachers 
and counselors were highly invested in acknowledging 
the problem and actively participating in the solution. 
For this reason, we knew we were looking for a 
program where our school staff would be heavily 
involved in the process. 

The Guide highlighted that both Second Step and PATHS 
are programs implemented in the school by teachers 
and counselors (see Table 2A). The advisory panel 
agreed that we appeared to be on the right track. 


Considering intervention characteristics 

The Guide helped to highlight quickly differences 
between the two programs (see Table 3A). The PATHS 
program did not include a family component. From 
discussions among the advisory panel members, 
it became clear that some members of the panel 
believed that changes in children's behaviors would be 
more successful if families were involved in the process. 
However, the advisory panel as a whole was not ready 
to exclude a program on this one component. 

The final concern of the advisory panel was that a 
program would be able to track fidelity and provide 
outcome measures. The advisory panel wanted to 
be sure the program would be implemented both 
according to design and the program's positive, 
measured, effect on the children. The Guide showed 
that both Second Step and PATHS designated both the 
fidelity and outcome measures we wanted (see Table 
5A). It appeared to all on the advisory panel that two 
programs could potentially be used in their school. 

To continue the decision process, the advisory panel 
decided they would use the program descriptions 
provided in the Guide to gain more detailed information. 

Indepth review of multilevel/prevention programs 

First, we reviewed the Second Step program. A few 
panel members noted that Second Step was implemented 
using curriculum kits. It was also noted that in regard 
to training, each participant receives a comprehensive 
Trainer's Manual, CD-ROM, and a set of four staff 
training videos. 
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Professional development credits are available for 
completion of the regional Second Step Training for 
Trainers. The panel recognized that this feature would 
appeal to the teaching staff. The advisory panel noted 
too that the cost for materials varies according to the 
curriculum kits purchased and ranges from $1 59 to 
$289, but that volume discounts are available for 
orders over a certain size. Finally, the advisory panel 
noted that Second Steps has a funding specialist on 
staff to provide information on up-to-date grant 
announcements and funding opportunities. The 
advisory panel decided to explore the developer's 
website, as this information was provided as part 
of the program description. 

Next, the panel reviewed the PATHS program. The Guide 
showed that PATHS is a 5-year program. The PATHS 
curriculum provided a manual with specific instructions 
and developmentally appropriate lessons. The advisory 
panel noted that the curriculum materials ranged from 
$100 to $679, higher than Second Step. Unlike Second 
Step, the PATHS curriculum was available in Spanish. 

The advisory panel considered how useful this would 
be for the students. PATHS, however, did not provide 
any formal readiness instruments, something the panel 
felt would be important for the teachers. The program 
description provided the developer's contact 
information. The panel decided to contact the 
developers to have their remaining questions clarified. 

Further directions 

With the help of the Guide for Selecting and Adopting 
Evidence-Based Practices, the advisory panel gathered 
information that considered the needs of the community, 
the population served, and the fit of the intervention 
with the agency, the families and the youth. From here, 
the panel would further explore the two programs, 
decide on one, and then bring the program to the 
school board, the school staff, and the parent 
association for review. 


Case Illustration 2: Intervention 

A recent needs assessment revealed that an urban 
community just outside of a major U.S. city saw an 
increase in disruptive behaviors of adolescents. Examples 
of these include juvenile arrests, underage substance 
use, and lower school performance. As a result of this 
assessment, the community buzzed about the need 
for some type of intervention. The director of a major 
mental health clinic, Dr. Cook decided to bring together 
an advisory panel to help consider a number of possible 
interventions that could be implemented to address the 
growing need of adolescents in the community. 

The advisory panel that was created consisted 
of a number of different voices in the community: 

■ The state mental health director; 

■ The directors of the local child welfare agency; 

■ Juvenile justice agency; 

■ The community mental health clinic director; 

■ Clinicians from the mental health center; 

■ Area high school principals; 

■ Representative of family organizations; 

■ Families; and 

■ Youth. 

Dr. Cook decided to use the resource, A Guide for 
Selecting and Adopting Evidence-Based Practices for 
Children and Adolescents with Disruptive Behavior 
Disorder, as a tool for selecting an evidence-based 
practice. As the advisory panel met, it soon become 
apparent that a number of decisions were to be made 
to narrow down the information in the Guide to just 
one program. 

The first decision was whether to implement a 
prevention program or an intervention. The advisory 
panel noted that their interest lay in addressing the 
needs of children with identified disruptive behavior 
disorders. Therefore, the panel focused only on the 
programs marked as Interventions in the Guide. 
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Considering the population of interest 
and desired outcomes 

The advisory panel first explored the Guide (see Tables 
1A and B) to compare the identified population of the 
interventions with their own community needs. Knowing 
that the focus was on late adolescence, the panel did not 
consider any program that did not extend to youth age 
18 years. Neither did it consider any program that had 
not included Hispanic populations, as the community 
was predominantly Hispanic. With these limitations, the 
advisory panel was considering four interventions: Brief 
Strategic Family Therapy™, Functional Family Therapy, 
Mentoring, and Multidimensional Treatment Foster Care. 

The next choice to make was to identify the specific 
outcomes that the panel believed were important to 
address. In the Guide (see Table 3B), the outcomes 
from the Brief Strategic Family Therapy and Mentoring 
were of most interest. 

Considering the characteristics of agency 

Next, the advisory panel reviewed the intervention 
characteristics in the Guide (see Table 3B). The 
advisory panel decided to exclude Mentoring from 
further consideration because of the lack of a family 
component. Perhaps, the advisory panel considered, 
Mentoring could be an intervention that was added 
to the community, but not necessarily through the 
community mental health clinic. 

Considering the implementation process 

Now only considering Brief Strategic Family Therapy™ 
and Multisystemic Therapy, the panel explored the 
cost and length of training highlighted in the Guide 
(see Table 4B). A significant difference in cost existed 
between the models; this was not, however, an 
immediate concern or reason to exclude either 
program yet. 


Further directions 

The panel decided that it would read the individual 
program descriptions in more detail, and meet again 
to discuss the idea of implementing one or both of 
the programs at the clinic. Two panel members also 
volunteered to contact the program developers to 
gather more detailed information. The advisory panel 
would meet again to discuss the information and 
continue the process of selecting an intervention 
or interventions to implement in the community. 


Hints on Understanding 
Research Study Designs 

Some readers want a better grasp of research so 
that they have a clearer understanding of the ways 
in which the EBPs in this KIT were evaluated. 
This section provides a very brief overview. 

How do we know that the evidence for one 
intervention is better or stronger than for another 
intervention? The answer depends on the way the 
studies are designed and conducted. As a study’s 
results show that an intervention is effective and 
has achieved the desired outcome, it is also 
important to verify that people who received the 
intervention did not improve for some other 
reason. Research designs do this by controlling 
for variables that could contribute to the person’s 
improvement. For example, people can get better 
because an illness simply took its course, or 
perhaps their health improved because of 
additional care they received beyond the 
intervention itself. Others might appear to get 
better because of the way the study was designed 
and conducted. A study that is not well designed 
might incorrectly lead to the conclusion that the 
intervention made people better when that was 
indeed not the case. 
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Some research designs compare one group of 
people who received the intervention (the 
experimental group) with another group that did 
not (the control group). If most people in the 
experimental group improved while those in the 
control group did not, one conclusion might be 
that the intervention is effective and that people in 
the experimental group that received the 
intervention are better as a result. 

Can we say that? Is such a conclusion valid 0 Not 
necessarily; significant differences might have 
existed between the two groups in terms of age, 
sex, ethnicity, or other characteristics that could 
affect the outcome. 


Researchers use specific designs called randomized 
control trials (RCT) to address these alternative 
explanations. As shown in Figure 2, an RCT is a 
study in which there are two groups: one treatment 
and one control group. The treatment group 
receives the treatment under investigation, and the 
control group receives either no treatment or 
standard treatment. An important feature of RCT 
studies is that children and adolescents are 
randomly assigned to each group so that each 
group has a similar sample population. 


Figure 2: Randomized Control Trial (RCT) Design 


Treatment Group Followup 



Random Assignment 


Note: Chart is adapted from "Understanding Research Study Designs" by the University of Minnesota 
Bio-Medical Library, 2010, Minneapolis, MN: Author. Copyright 2010 Regents of the University of 
Minnesota and the Health Sciences Libraries 
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Randomized control trials are considered to 
have stronger “proof’ than other types of studies. 
When many such RCT studies are conducted— 
in different locations, by different researchers, 
in settings that resemble the real world—the 
evidence that the intervention is effective builds 
and is increasingly corroborated. These interventions 
are the ones that obtain the highest rating in terms 
of evidence. 

Other studies may have been conducted by just 
one group of researchers or in just one place. 

These interventions have less evidence, but may 
still be effective. 

Other types of studies, termed quasi-experimental, 
are similar to the randomized control trial, except 
that there are no random assignments to the 
different groups. This type of study is still useful 
in determining the effectiveness of an intervention, 
but the evidence resulting from this type of study 
is not as strong as a randomized control trial. 


The important point here is that some 
interventions have more — or higher—levels 
of evidence than others. These levels of evidence 
are based on the selection of study designs and 
the number of times the interventions have 
been evaluated as successful. Different schemes 
exist to describe such levels of evidence. The 
American Psychological Association has a 
hierarchy of the levels of evidence. The National 
Institute of Mental Health (NIMH) also has an 
approach to such levels. These can be accessed 
at http://www.apa.org and http://www.nimh.nih.gov, 
respectively. 

In the booklets of this KIT, you Mil see many 
references to the level of evidence found for the 18 
ERPs covered in this KIT. All 18 ERPs have at 
least a moderate to good level of evidence support. 
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